From Label Smoothing to Label Relaxation

Regularization of (deep) learning models can be realized at the model, loss, or data level. As a technique somewhere in-between loss and data, label smoothing turns deterministic class labels into probability distributions, for example by uniformly distributing certain part mass over all classes. A predictive model is then trained on these distributions as targets, using cross-entropy function. While this method has shown improved performance compared to non-smoothed cross-entropy, we argue that use smoothed though still precise distribution target questioned from theoretical perspective. an alternative, propose generalized called relaxation, in which set probabilities represented terms upper distribution. This leads genuine relaxation instead distortion, thereby reducing risk incorporating undesirable bias process. Methodically, minimization novel type function, suitable closed-form expression optimization. The effectiveness approach demonstrated empirical study image data.